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Abstract 


Despite hackathons having a large potential as an instructional strategy for stimulating student 
engagement and collaboration in teams, their application as a formal teaching approach and integration 
into curricula in higher education is less prevalent. This paper reports on applying the format of an online 
hackathon as a collaborative and inclusive instructional design approach, in which university students 
from different countries and different academic backgrounds came together for two days to jointly work 
on solutions in the field of digital education during a hackathon challenge as a sub-event of a larger, 
European hackathon event. The paper describes the design of the hackathon challenge, its integration 
into curricula of the university courses, the implementation, the results of in terms of solutions submitted 
by student teams and the insights from the exploratory study drawing on the observations of the mentors 
and the results from an online survey with 57 students. The results show that students who participated 
in the hackathon challenge not only enjoyed the learning experience but also recommended to include 
online hackathons in higher education as a voluntary activity and as a way to train and find talents. The 
participants valued the inclusiveness of the hackathon and enjoyed designing new solutions for 
problems with real-world applications. However, the study revealed some challenges in forming 
international teams and in collaborating at distance during the hackathon. Therefore, the paper outlines 
possible design solutions for preparing students for international teamwork during hackathons. 
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1 INTRODUCTION 


The COVID19 pandemic has imposed severe limitations both on communication between students and 
teachers and on international exchanges among students and educators. While instructional designs 
during the pandemic have explored a diversity of methods going beyond traditional online lectures, 
including formats such as flipped classroom, co-teaching and problem-based learning, only a few 
authors so far have reported on applying online hackathons as an instructional design approach to 
supporting education during the pandemic and beyond. For example [1] reported on an online, 
educational hackathon during the global pandemic of COVID19 as a method to develop fundamental 
professional skills of students in software engineering. The key learning results of such an educational 
hackathon included communication and collaboration skills, initiative, creativity and innovation [1]. The 
case study by [2] about the virtual hackathon during the COVID-19 pandemic showed how hackathons 
can be designed as a cooperation between local universities, industry, and government to foster 
innovation based on digital entrepreneurship. The hackathon EUvsVirus led by the European Innovation 
Council and described by [3] shows how hackathons can be used during the COVID-19 pandemic to 
enhance open innovation, leverage distributed knowledge to solve specific problems and develop 
innovative solutions. These and other examples draw on positive experiences in using hackathons in 
education before the pandemic. Among others, in 2017 [4] described how a weekend hackathon 
integrated into a semester-long undergraduate course facilitated inquiry-based learning. Later in 2019, 
[5] explored differences between curricular and traditional hackathons in the context of higher education 
and pointed out that there is a shortage of published literature on “curricular” or “formal” hackathons. 
The report by [6] touches upon the importance of hackathons as an alternative approach for practicing 
and learning in higher education and highlights key positive aspects of educational hackathons such as 
peer learning, improvements in problem-solving, project management skills, as well as high levels of 
engagement and enthusiasm. Despite hackathons having a large potential as an approach to promoting 
engagement and openness across organizational boundaries, and as an instructional strategy for 
stimulating student engagement and collaboration in (international) teams, systematic reports on the 
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design and the application of online hackathons in higher education as part of the curricula, especially 
during COVID-19 pandemic are barely available. 


This paper reports on applying the format of an online hackathon as a collaborative and inclusive 
instructional design approach in the context of higher education, in which university students from 
different countries and different academic backgrounds came together for a few days to jointly work on 
innovative solutions in the field of digital education. The online hackathon described in this paper was 
embedded in a larger hackathon ecosystem and organised as a sub-event (a hackathon challenge) at 
the DigiEduHack2021', an annual series of online, blended and in-person hackathon events, initiated 
by the European Commission's Digital Education Action Plan. The online hackathon challenge 
described in this paper was called “Emerging technologies for holistic learning’? and aimed at 
developing innovative solutions using emerging technologies for holistic and engaging learning 
experiences. The participants in this hackathon challenge were students from universities in different 
countries worldwide. The key research question addressed in this paper is how to design online 
hackathons to enhance student engagement and collaboration in international student teams as part of 
higher educational curricula in contexts such as the COVID19 pandemic. The paper reports on the 
design of the hackathon challenge and on the results from an exploratory study based on an online 
survey with 57 students, which evaluated the online hackathon challenge as an instructional approach 
from the perspective of the participants. 


Following this introduction, Section 2 describes the design of the hackathon challenge, including its 
integration into curricula and the implementation of the challenge during the DigiEduHack2021 event in 
November 2021. Section 3 presents the results and insights from the online evaluation survey. Section 
4 includes conclusions related to the potentials and challenges of online hackathons in the context of 
higher education. The paper ends with suggestions for integrating hackathons into university courses 
together with possible areas for future design and research. 


2 DESIGN 


A hackathon, a compound of “hack” and marathon”, is a computing event in which participants 
collaboratively, intensely and rapidly prototype new solutions over a short period of time, usually one to 
two days [5]. Hackathons are focused on producing results such as concepts, prototypes or new 
software/product features, through problem solving and playful experimentation in teams in a limited 
amount of time [7]. This section describes general design options for hackathons and presents the 
design considerations for the DigiEduHack2021 hackathon challenge. 


2.1 Hackathon design options 


When designing a hackathon, there is wide range of diverse design options, themes, methods, formats 
and audiences to choose from. Some key design options described here provide a foundation for the 
description of the design of the DigiEduHack2021 hackathon challenge in the next section. 


Hackathons have been popular as coding events in the ICT sector and focused on bringing 
programmers, developers and designers together to collaborate on software projects in a short period 
of time [8]. Traditionally, hackathons have been used to promote innovation in technology development 
and/or new uses for existing technology [5]. Over the time, different kinds of hackathons emerged to 
meet different objectives and cater to the needs of different audiences, e. g. data-dives, code-fests, 
code-jams, code-camps, hack-days, edit-a-thons, or game-labs. Beside the traditional coding 
hackathons, which have been largely driven and sponsored by large corporations [5], new forms of non- 
coding hackathons emerged in recent years. These have included educational hackathons, organised 
by (public) universities and a diversity of informal (non-curricular) and formal (curricular) hackathon 
events, which have been designed a form of event-based education, similar to other formats such as 
BarCamp, World Café or TeachMeet [9]. 


Moreover, while traditional hackathons take place in dedicated venues and participants remain primarily 
at these venues with limited breakaways [5], more and more hackathons started taking place online or 
combine online and onsite into a blended experience, just like the example of the DigiEduHack2021 
shows. The comparison of online and onsite hackathons by [7] shows that both formats thrive on 


' https://digieduhack.com/en/ 
? https://digieduhack.com/en/berlin-emerging-technologies-for-education 
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spontaneous social interactions between the participants, and that these social interactions can be 
stimulated both onsite and online. While participants in onsite hackathons report on organisational and 
schedule-related issues as usual obstacles, participants in online hackathons name working alone at 
home both as an advantage (e. g. undisturbed work) and an obstacle (e. g. insufficient equipment) to 
participation [7]. 


Further design options for hackathons are related to the openness of the event. Hackathons can be 
designed as closed or open events [5]. While closed hackathons are internal to organizations and focus 
on a new product (feature) or innovations for a specific company, open hackathons are organized as 
public or civic events, encourage broad attendance and participation and focus on a specific topic or a 
technology such as healthcare or wearable devices [5]. 


Next, hackathons can be designed as short events spanning one to two days or as extended events. 
The design of a short, weekend-long hackathon is described in [4]. This example is rooted in the 
constructivist learning theory and applies the method of inquiry-based learning (IBL) with a problem or 
task serving as a catalyst for student engagement and participation [4]. The design of an extended 
hackathon model spanning one to two weeks and integrating the fast-paced, collaborative prototyping 
and business model development with an innovation curriculum in the field of healthcare innovation is 
described in [10]. The extended hackathon model consists of three elements — short seminars, field trips 
to local hospitals, and iterative, hands-on prototyping supported by mentors [10]. 


Furthermore, hackathons can be designed as stand-alone events or as part of a larger hackathon 
organisation or ecosystem. An example of hackathons embedded in a larger organisation are the 
hackathon event “HoloHack 2017” and the GameJam event 2018 hosted at the university-based 
Learning Factory [9]. An example of the concept of a hackathon ecosystem is provided by [11] and is 
related to the technology transfer in higher education. 


2.2 DigiEduHack hackathon challenge design 


Based on the design options listed above, the hackathon challenge “Emerging technologies for holistic 
learning” described in this paper was designed as a non-coding, short (two days), educational, open, 
curricular, online event embedded in a larger hackathon ecosystem of the DigiEduHack2021. The 
hackathon challenge was a non-coding event as it aimed at developing innovative educational concepts 
for holistic and engaging learning with support of emerging technologies. The envisaged outcomes were 
neither program codes nor software features, but rapidly prototyped concepts visualised with the help 
of graphic representations such as diagrams or mock-ups. The event lasted only two days, took place 
fully online, and was open for all students to participate. At the same time, the hackathon challenge was 
integrated into the curricula of two courses at the two universities which facilitated the hackathon, i. e. 
Berlin University of Applied Sciences in Germany and the Academic College Tel Aviv-Yafo in Israel. The 
integration into the curricula of these two courses was twofold. First, the participation in the hackathon 
was thematically and organisationally integrated into the curricula, e. g. the topic of the challenge was 
part of semester projects, kick-off sessions took place during regular class times. Second, participating 
students received (bonus) points for participation and/or submission which counted for the final grade. 
In this sense, the hackathon challenge was a blend of an informal (non-curricular) and formal (curricular) 
approach, depending on whether the participants came from the facilitating universities or from any other 
higher education organisation. 


Altogether over 140 participants from all over the world formed the total of 28 international teams to 
participate in the hackathon challenge. The participating students had to sign up for the event on the 
DigiEduHack2021 platform before the event. Since participation was open to all students and students 
from countries outside Europe, e. g. Kenya and India, joined the challenge and worked in international 
teams. The hackathon challenge was facilitated by seven persons from altogether six universities, i. e. one 
host (first author of this paper), five mentors (including the second author of this paper) and one community 
manager. The role of the host was to set up the challenge and lead the event. The role of the mentors was 
to support the teams in relation to the content of the challenge and working/collaboration methods in 
international teams. The role of the community manager was to support the teams in relation to 
organisational and technical issues. A dedicated workspace in Slack with a channel for each team and for 
diverse topics was used to streamline communication (cf. Fig.1). 
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Q Search DEH2021-BERLIN @ 
DEH2021-BERLIN ~ # general - E> ee 


Tuesday, November 2nd ~ 


PRB llona Buchem (host) 11:03 AM 
E Dear Participant, 
Welcome to the Challenge "Emerging Technologies for Holistic Learnig". Happy to have you on board! 
This challenge starts next week and we (host & mentors) are all excited to meet you and learn about your first ideas. 


Here are 3 important points for you before we start next week: 


1. The first milestone for this challenge is to submit the first idea by 8th November, 22:00 Berlin time. Please submit your 
first idea using this online formular: 
https://docs.google.com/forms/d/e/1FAlpQLSd-9Ck4esJFSS335D06duOpxN YouRcr162JyOmbYuNYIKYybg/viewform 


2. All the information about how to prepare and submit the first idea and also all the steps of how we will collaborate in 
this challenge are included in this presentation: 
https://docs.google.com/presentation/d/1tHBLtSbLKgTGbxNBvyONTwBknmEdwPwJsGbGrja2ts/ 

Please read through the slides to have a good understanding of how this challenge works. 


3. To communicate during the challenge we will use SLACK as a tool. Please make sure to join our workspace: 
https://join.slack.com/t/deh2021-berlin/shared_invite/zt-y804xelt-fYSd1IbhBUSKuE90qVX4dg 


If you have any questions, please use Slack to communicate with us. 

The Mentors will support you in the development of the solution. Our Community Manager is here to support you with 
organisational and technical issues. 

Looking forward to hearing from you! 

ranslation Best from Berlin, 


ogramming 


Figure 1. Screenshot from the DEH20201 challenge Slack showing teams channels. 


Additionally live meetings in the web conferencing system Zoom were organised for all participants of 
the hackathon challenge. Moreover, each team organised their own meetings in Zoom, Google Meeting 
or WhatsApp and used other tools for collaboration such as Google Drive or prototyping tools such as 
Microsoft PowerPoint, Axure, Figma. 


The challenge started on the day before the official hackathon with the submission of the first idea. To 
submit the first idea, participants were provided an Idea Building Canvas and the idea was uploaded by 
participants via an online submission form, created using Google Forms. Submissions had to be made 
in teams. Based on the initial submissions, the host created channels for each team in Slack. The 
hackathon challenge started a day later, on the first day of the hackathon, with the morning kick-off 
session in Zoom. All participants were invited to join the session and at the same time the session took 
place as a regular class in the course at Berlin University of Applied Sciences. The aim of the kick-off 
was to provide a short overview of submitted ideas, so that participants, who did not submit any idea 
could join existing teams. Most of the time during the first day of the hackathon challenge was dedicated 
to self-organised work in teams. Participants were supported by the facilitators in forming teams, 
choosing the idea/team to work with, contacting team members, and agreeing on how to work on the 
solution. To work on the solution, another template -the Solution Canvas — was provided based on the 
general outline proposed by DigiEduHack2021 organisers for all hackathon challenges. The deadline 
for submitting final solutions was by the end of the second day 2. The second day started with a new 
kick-off session in zoom. All teams were invited to attend this session to share their current status, get 
feedback and ask any questions. The middle part of the second day was again dedicated to self- 
organised work in teams using the Solution Canvas. The host, the mentors and the community manager 
supported all teams in working on the final submission. The second day of the challenge ended with a 
closing session in Zoom which was open to all participants and at the same time took place as a regular 
class in the course at the Academic College Tel Aviv-Yafo in Israel. The closing session started with a 
short wrap-up of the two days. Then teams presented their solutions and received feedback. Finally, 
participants were invited to fill in the online survey in Google Forms. After the closing session in Zoom 
participants had a few hours time to submit their final solutions to the general DigiEduHack submission 
system. After the hackathon event, the host and the mentors chose three best solutions and the 
challenge winners were designated to the DigiEduHack steering group, which had the task to designate 
10 to 12 finalists. The finalists from all challenges organised as part of the DigiEduHack are put up fora 
public vote. The 3 finalists with the most votes receive the DigiEduHack 2021 Global Award, which 
includes the prize of 5000€ and the status of the DigiEduHack Global Ambassador. 


From 28 initial teams, 21 teams submitted a final solution to the DigiEduHack system. The winner of the 
challenge “Emerging technologies for holistic learning” was the team/solution named “VR & ICT for 
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empathy”’. These criteria used by the jury to assess the quality of the challenge solutions included: (a) 
innovative idea and a meaningful challenge the project is trying to solve, (b) quality of the description of 
the solution, (c) understanding the needs of end users. All submitted solutions were published on the 
DigiEduHack challenge website and are publicly available‘. 


3 METHOD AND RESULTS 


An exploratory study was conducted to evaluate the online hackathon challenge as an instructional 
design approach. The virtual hackathon was evaluated using a quantitative approach and the data was 
collected with an online survey. The key research question guiding the study was related to the 
exploration of a hackathon as a learning environment and its instructional design. 


3.1 Study tools 


The online survey was designed using the online survey tool SurveyMonkey. The survey included three 
groups of 35 questions in total: Part | Socio-demographic data (7 questions); Part Il Communication (14 
questions); Part IIl Learning Design of the hackathon (14 questions). Items in Part | Socio-demographic 
data with standard information related to the country of origin, age, gender, education level, field of 
study, as well as further items related to previous international experiences such as working in 
international teams, previous experiences in communicating in the English language and previous 
participation in hackathons. Part Il Communication was used to evaluate the learner experience in the 
hackathon as a learning environment. These items were derived from the User Experience 
Questionnaire (UEQ), which is an end-user questionnaire to measure user experience in the context of 
software products [12] and was already adjusted to the context of the hackathon [13]. Items in Part III 
Learning Design of the hackathon were designed by the authors of this paper to capture perceptions of 
students related to the specific elements of the instructional design of the hackathon such as the 
framework, the reason to participate and the collaboration. 


3.2 Study sample 


Altogether 146 students in 28 groups participated in a two day online hackathon. The online survey was 
conducted with participating students at the end of the hackathon. The students were given approx. 15 
minutes to fill in the questionnaire before the end of the last meeting. 57 students (39%) participated in 
the online survey. Table 1 gives an overview of the characteristics of the study sample. 


Table 1. Characteristics of the study sample (n=57). 


Socio-demographic categories Study sample 


75.4% Israel (n=43) 
19.3% Germany (n=11) 
3.5% India (n=2) 
1.8% Italy (n=1) 
Age M = 26.7, SD = 5.65 
52.6% female (n=30) 
47.4% male (n=27) 
Previous experience International experience: M=3.00, SD=1.64 
(1 = very weak, 5 = very strong) | English language experience: M=4.53, SD=1.24 
Bachelor students = 49.7% 
Education level Master students = 35.1% 
Other = 15.2% 
None - 79% (n=45) 
One to two - 12.3% (7) 
Three and more - 8.7% (n=5) 


Country of origin 


Gender 


Number of hackathons 
participated in the past 


3 https://digieduhack.com/en/solutions/vr-ict-for-empathy 
* https://digieduhack.com/en/year/2021/2021-solutions 
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3.3 Results from the participants’ survey 


In this section we describe the results of the online survey with 57 students from Israel, Germany, India 
and Italy (53% female, 47% male). The data analysis was an exploratory and iterative process. To 
analyse the survey data, IBM SPSS Statistics software (version 28) was used to code, categorize, and 
conduct the data analysis. 


The number of final submissions can be considered as one of the key outcomes of the hackathon: Out 
of the organized 28 groups who participated in the hackathon, 21 teams (75%) submitted their final 
solution and 16 teams (59%) created a prototype such as paper prototype, presentation or an interactive 
mock-up. When asked “What do you think about the hackathon framework?”, 41% of the students 
positively assessed the duration of the challenge, while 35.9% indicated that the duration of two days 
was too short (cf. (Fig. 2). 74% of the students stated that the number of participants in their group was 
enough (M=4.46, SD=3.51). 56.4% felt that the amount and the duration of the joint online Zoom 
meetings during the hackathon were enough, and 47.37% of students stated that they received just 
enough support from their mentors. 


100% 
90% 
80% 
70% 
60% 
50% 
40% 
30% 
20% 
10% 


0% 
The duration of The number of the online zoom the support of 
the hackathon (2 participants in meetings the mentors 
days) your group 
cn Too little EY Enough @& Too much 


Figure 2. Students’ responses to the question “What do you think about the hackathon framework?” (n=39). 


When prompted to mark what were the main reasons to attend the hackathon (cf. Table 2), the most 
common answer was “because | was told to do so by the teacher’ (53.85%). Still, 33.33% of the students 
stated that curiosity to try new things also led them to participate. Almost half of the students (46.15%) 
said that neither the monetary prize nor developing own ideas or prototypes was a good enough reason 
to participate (43.59%). Only 25.64% of the students stated that the reason to attend was to improve 
their knowledge, learn about new tools and contribute to improving education. 


Table 2. Reasons to attend the hackathon (n=39). 


Reasons to participate Not Important | EitherWay | Important 
Because | was told to do so by the teacher 12.82% 33.33% 53.85% 
To network and meet new people 38.46% 38.46% 23.08% 
To learn about new tools, software and technologies 38.46% 35.90% 25.64% 
To contribute to improving education 35.90% 38.46% 25.64% 
To do something good for the community 38.46% 38.46% 23.08% 
To win the prize 46.15% 30.77% 23.08% 
To promote myself 35.90% 41.03% 23.08% 
To improve my professional career prospects 35.90% 41.03% 23.08% 
Curiosity about trying new things 35.90% 30.77% 33.33% 
They're fun! 41.03% 33.33% 25.64% 
| like building prototypes of my ideas 43.59% 35.90% 20.51% 


8647 


The students met online with each other during the two day hackathon a multiple times and 
communicated directly in international groups. Part II of the online survey was derived from the User 
Experience Questionnaire (UEQ), an end-user questionnaire used to measure user experience [10]. 
The items from UEQ were adjusted in wording and number to the context of the hackathon. The adapted 
UEQ items were fit for purpose and measured learner experience in the learning environment of the 
hackathon [13]. The highest values were reached for the aspects friendliness (M=5.18, SD=1.47), 
excitement (M=4.74, SD=1.12) and joy (M=4.66, SD=1.38). The lowest values were reached for the 
value (of the hackahton experience) (M=3.57, SD=1.65) and clarity (M=4.03, SD=1.58). The results of 
the descriptive analysis are summarized in Table 3. 


Table 3. Communications aspects’ descriptive statistics (Scale 1-5 with 1=low, 6=high) (n = 47) 


UEQ items Mean SD 
Value 3.57 1.65 
Motivation 4.32 1.41 
Excitement 4.74 1.12 
Joy 4.66 1.38 
Practicality 4.44 1.31 
Interest 4.10 1.57 
Friendliness 5.18 1.47 
Understanding 4.28 1.82 
Ease 4.21 1.70 
Clarity 4.03 1.58 
Organization 4.40 1.50 


When asked whether students would recommend online hackathons in higher education, 56.4% of the 
students recommended to include hackathons as part of course curricula. However, most students 
recommended not to include hackathons as a part of course assessment procedures but rather as a 
voluntary activity without grades or merits (71.8%) and as a way to train and find talents (58.9%). 


The students also stated that they enjoyed designing new solutions (56.4%) and working on a problems 
that have real-world applications (56.4%). When asked “What are the key outcomes from the hackathon 
for you?”, 33.3% of the students ranked “Learning how hackathons work” as the main outcome, while 
25.6% of the students ranked “Higher awareness of digital education” as the key outcome. 


4 CONCLUSIONS 


The key research topic guiding the study was the exploration of an online hackathon as an instructional 
design approach. The results of the exploratory study show that the design of the hackathon was 
effective in the sense that 75% of the groups submitted a final solution at the end of the two-day 
hackathon challenge. This may be due to the structure of the hackathon, which was designed as a short 
event with clear milestones and templates that supported the creation and the submission of group 
solutions. Another reason could be the help of the mentors and the satisfaction of the participants with 
the quantity of the support from the mentors. Other results indicate that the instructional design of the 
hackathon was also effective in the sense of user experience. Students not only enjoyed the learning 
experience during the hackathon but also recommended to include online hackathons in higher 
education as a voluntary activity and as a tool to train and find talents. 


Surprisingly enough, it was the first hackathon for most students (79%) although their field of study in 
information systems and digital business is closely related to the traditional application field of 
hackathons. Probably, the students were not given the opportunity to participate in a hackathon before 
but possibly also did not actively look for and participate n a hackathon out of their own interest. Most 
students indicated that the main reason for participating (54%) was due to a compulsory requirement in 
the course. This insight indicates that it is even more important to expose the students to the framework 
of a hackathon early in their studies in order to give students the opportunity to experiment and learn 
the mechanism and the value of a hackathon. As described by [4], hackathons integrated into a 
semester-long undergraduate course can be also used to facilitate inquiry-based learning. 
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As a result of this exploratory study, we formulate several recommendations for the instructional design 
of hackathons and integrating hackathons into higher education course curricula: 


1 


First, since online hackathons are an engaging format for collaboration, especially in international 
settings and in times when travel is not possible, we recommend to include hackathons as an 
instructional method in study programs. As indicated by [7], online hackathons provide an 
advantage of flexible participation from home. 


Second, as far as an instructional design of a hackathon is concerned, we recommend exposing 
students to the format of a hackathon during their studies as a part of the regular course 
curriculum, so the students can learn how to face the challenge of remote collaboration in 
international teams, which may be an important part of there work life at some point in time. 


Third, from our experience, students are not available for two days of full participation in a 
hackathon during their semester, due to their other commitments including study and work. 
Therefore, we recommend stretching the challenge in time for at least two weeks with shorter 
spans of time per day for team collaboration. 


Furthermore, we recommend planning an introductory meeting with all students prior to the 
hackathon, in which students can learn about the hackathon as an event format, about its value 
for work life, innovation and group collaboration, as well as about examples of successful 
hackathons from different fields. 


Finally, to enhance interaction between groups, we recommend creating and using a peer 
feedback rubrics or score sheet for students to assess the work of their peers. Peer feedback in 
hackathons can be used in addition to the evaluation of mentors or judges and can possibly 
enhance involvement and commitment of students. 


Following up on the research by [1], possible areas for future research include the exploration of what 
skills students develop during hackathons (e.g., problem solving, digital skills, collaboration) and how 
students may apply these skills at their work, e. g. after graduation. 
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